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ABSTRACT 


i 

A  strictly  proper  scoring  rule  or  admissible  probability 
measurement  procedure  (APMP)  is  a  mathematical  device  that 
allows  a  decision-maker  to  "score"  probabilistic  forecasts 
made  by  experts  once  the  outcome  is  known.  The  expected 
score,  as  seen  by  the  forecaster,  should  be  maximized  when 
the  forecaster  states  his  true  beliefs  so  that  he  will  be 
encouraged  to  do  so.  Applications  to  subjective  probability 
assessment  will  be  given. 


THE  CHARACTERIZATION  OF  STRICTLY  PROPER  SCORING  RULES 

IN  DECISION  MAKING 
by  Eduardo  Haim 

1.  INTRODUCTION 

A  convenient  way  to  express  beliefs  on  future  outcomes  is  to  give  a  forecast  weighted  by 
a  probability  distribution.  For  example,  in  weather  forecasting,  the  forecaster  attaches  probabil¬ 
ities  to  the  events  that  snow,  rain  or  sun  will  occur.  These  probabilistic  forecasts  which 
describe  the  beliefs  of  an  expert  may  be  used  by  decision  makers  in  their  analysis  of  a  problem. 
It  is,  therefore,  important  to  be  able  to  "score"  the  expert  based  on  past  performance  in  order  to 
attach  a  "degree  of  credibility"  to  the  forecast.  It  is  also  important  to  get  a  forecast  which  is 
representative  of  the  expert’s  "true"  beliefs.  The  expert  should  not  feel  that  by  stating  a  hedged 
forecast  he  will  get  a  higher  score. 

A  strictly  proper  scoring  rule  or  admissible  probability  measurement  procedure  (APMP)  is 
a  mathematical  device  that  allows  the  user  (or  decision  maker)  to  score  the  expert’s  forecast  in 
such  a  way  that  the  expert  will  be  "encouraged"  to  state  his  "true"  beliefs.  These  concepts  will 
be  defined  more  rigorously  in  Section  2.  In  Section  ?,  the  theory  that  will  help  in  the  analysis 
and  construction  of  scoring  rules  is  presented.  In  Section  4,  the  results  of  the  previous  section 
are  used  to  construct  a  strictly  proper  scoring  rule  for  a  given  model.  The  summary,  conclu¬ 
sions  and  topics  for  further  research  are  in  Section  5. 

This  paper  deals  exclusively  with  forecasts  weighted  by  discrete  probability  distributions, 
the  continuous  case  will  be  treated  in  another  paper.  The  main  reference  to  this  work  is  a  1971 
paper  by  L.J.  Savage  [3]  on  the  elicitation  of  personal  probabilities.  Application  of  scoring  rules 
to  weather  forecasting  and  some  desirable  properties  of  scoring  rules  are  presented  in  Refer¬ 
ences  (41  and  [51. 

An  important  aspect  which  is  often  overlooked  in  technical  papers  is  stating  the  reasons 
why  a  given  technique  is  useful.  Some  applications  of  scoring  rules  are: 


Elicitation  of  subjective  probabilities; 

Evaluation  of  expert  opinion; 

Use  of  past  performance  as  represented  by  the  scores  to  weigh  diverging  opinions  among 
experts; 

Multiple  choice  tests  could  allow  for  other  than  absolute  forecasts  and  use  scoring  rules  to 
evaluate  the  results. 

The  reader  is  referred  to  the  paper  by  Savage  1971  [31  for  other  uses  of  scoring  rules. 


2.  DEFINITIONS  AND  EXAMPLES 

Expert  opinion  is  often  given  in  the  form  of  a  probability  vector.  For  example,  a  weather 
forecaster  can  give  a  forecast  of  snow,  rain  or  sunshine  as  a  vector  (/•t,r2,rj)  such  that 

i"\  +  r2  +  rj  «■  1; 

0  <  r,  <  1,  i  -  1,2,3  , 

where 

ri  is  the  probability  of  snow 
r2  is  the  probability  of  rain 
r  3  is  the  probability  of  sunshine. 

Once  the  outcome  is  known,  the  users  of  the  forecast  would  like  to  score  the  weather  forecaster 
depending  on  "how  close’  to  the  outcome  his  forecast  was. 

Definition  1: 

Given  n  possible  events,  n  €  Z+  (positive  integers),  a  forecast  may  be  represented  as  a 
probability  vector  r  such  that 

f  *  ('i . r„),  r  €  R" 

D  ^  r,  ^  1,  i  »  1,  ...,  n  , 

n 

i- 1 

Let  the  outcome  be  Z+.  Let  the  forecaster's  score  be  given  by  the  real 

value  Sk(r).  We  call  the  real-valued  mapping  S  from  (r,k)  to  Sk(r)  a  scoring  rule .  A  score 
may  be  thought  of  as  a  reward  or  as  a  penalty.  In  this  paper,  we  shall  think  of  it  as  a  reward  so 
the  forecaster  wishes  to  increase  his  score. 

Let  the  vector  ..,/>*)  ,  p  €  R*  denote  the  forecaster’s  true  belief  (judgment), 

where 

0  <  Pi  <  1,  /  -  1 . n ,  and 


Then  p,  represents  the  probability  the  forecaster  attaches  to  the  occurrence  of  outcome  /. 
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The  stated  probability  forecast  r  may  not  be  equal  to  the  forecaster's  true  belief 
represented  by  p.  However,  the  users  of  the  forecast  would  like  to  use  scoring  rules  that 
encourage  the  forecaster  to  make  his  forecast  correspond  exactly  with  his  true  belief.  That  is, 
to  make  /■— p. 

Definition  2: 

Let  S*(r)  denote  the  score  assigned  by  the  scoring  rule  5  when  outcome  k  occurs  and  r 
is  the  stated  probability  forecast.  Let  £S(r,p)  be  the  forecaster’s  expected  score  where  the  vec¬ 
tor  p  is  defined  as  above  and 

ES(r,p)  —  J  p*S*(r)  . 

*•1 

Then,  the  scoring  rule  5  is  said  to  be  strictly  proper  if 

£S(p,p)  >  ES{r,p)  V  p,  with  equality  if  and  only  if  r  «■  p  . 

In  other  words,  S  is  a  strictly  proper  scoring  rule  if  and  only  if  the  forecaster’s  expected  score  is 
maximized  uniquely  when  he  sets  r-p. 

S  is  a  proper  scoring  rule  if  p  maximizes  £S  but  not  uniquely. 


Example  1 :  Brier  Scoring  Rule 
Let, 

r  6  R",  the  stated  probability  forecast  vector. 
p  €  R",  the  forecaster’s  true  belief  probability  vector. 

Let  n  be  number  of  possible  outcomes  and 
k  €  Z*,  1  <  k  <  n,  the  outcome. 

Deflne 

Sk(r)  -  -(r  -  ek)  r/(r  -  ek) 

where 


1  in  position  k 
0  elsewhere 

ek  is  a  vector  of  dimension  n  which  indicates  that  outcome  k  has  occurred  and  /  is  the  identity 
matrix  of  dimension  n. 

We  wish  to  show  that  the  Brier  Scoring  Rule  is  strictly  proper.  Note  that 

£S(p,p)  -  ES(r%p)  —  (r  -  p)TI(r  -  p)  >  0  V  p,  with  equality  if  and  only  if  r— p  since  /  is 
positive  definite.  Then, 

£S(p,p)  ^  £S(r,p)  Vp,  with  equality  if  and  only  if  r  •  p  . 

Therefore,  the  Brier  Scoring  Rule  is  strictly  proper. 

Note  that  the  proof  would  be  the  same  if  /  were  replaced  by  any  positive  definite  matrix 
of  dimension  n.  In  this  way  the  family  of  quadratic  scoring  rules  is  generated. 

Example  2: 

Let  r  €  R  *  be  the  stated  probability  forecast  vector. 

Let  p  €  R*  be  the  forecaster’s  true  belief  probability  vector. 

Let  n  be  the  number  of  possible  outcomes  and  k  €  Z+,  1  <  k  <  n,  the  outcome. 

Define  Tk(r)~— eT\r-ek\,  a  scoring  rule.  ek  is  the  same  vector  as  in  Example  1  and  e  is  a 
column  vector  of  ones.  Both  have  dimension  n.  Suppose  n -2,  then 

Expected  score  -  £T(r,p) 

-  -tp,(ki  -  1|  +  l'2“  0|)  +  p2(\rx  -  0|  +  |r2-  1 1)1 

-  A2p\ri  +  2p2r  J  . 

£T(r,p)  is  maximized  by 

{(1,0)  $  P\>Pi 

-  }(0,1)  if  p\<pi ' 

So  the  scoring  rule  T  is  not  proper. 


*  *  jmxsn  **  - 
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When  a  scoring  rule  is  not  as  simple  as  the  ones  presented  in  the  above  examples,  it  is 
harder  to  prove  that  it  is  strictly  proper  through  the  expected  score  definition  of  the  property. 
In  Section  3,  a  theorem  is  presented  that  will  make  it  easier  to  demonstrate  that  a  scoring  rule 
is  strictly  proper  The  theorem  can  also  be  used  to  construct  strictly  proper  scoring  rules.  In 
Section  4,  the  problem  of  constructing  scoring  rules  which  better  represent  the  interests  of  the 
users  of  the  forecast  will  be  discussed. 

2.1.  The  Utility  Function  of  the  Expert  (Forecaster) 

From  a  decision  analysis  point  of  view,  the  expert  forecaster  should  be  maximizing  his 
expected  utility  when  he  stales  his  forecast.  It  is  technically  incorrect  to  say  that  the  forecaster 
wants  to  maximize  his  expected  score. 

If  the  utility  function  of  the  forecaster  is  a  real  valued  function  /  such  that 

ft  ft 

£  Sk(r)pk  <  £  Sk(p)pk  Vp,  with  equality  iff  r—p 
*-i  i 

U)  if  and  only  if 

£/(-S*(r))p*  <  £/(S*(p))p*  Vp,  with  equality  iff  r-p 
*-l  *-l 

then  maximizing  expected  score  is  equivalent  to  maximizing  expected  utility. 

Let 

g  :  ft  —  ft  be  such  that 
g(x)  ~  ax  +  b  a  €  ft,  a  >  0,  6  €  ft  . 

It  is  easy  to  show  that  condition  (/)  holds  for  such  a  function  g. 

Necessary  and  sufficient  conditions  on  /  for  (/)  to  hold  are  very  restrictive  [6].  How¬ 
ever,  since  (/)  holds  for  the  linear  increasing  function  g,  it  will  be  assumed  that  the  utility 
function  of  the  forecaster  is  piecewise  linear  and  increasing  in  the  score.  This  assumption  is 
not  too  restrictive  since  any  smooth  function  can  be  arbitrarily  approximated  by  piecewise  linear 
functions  and  the  utility  function  of  the  expert  should  be  increasing  in  the  score.  For  relatively 
small  changes  in  the  value  of  the  score,  the  utility  function  can  be  assumed  locally  linear. 
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Based  on  the  above,  maximizing  expected  score  will  be  assumed  to  be  equivalent  to  maximizing 
expected  utility. 

2.2.  Perfect  Forecasts 

Definition  3: 

A  forecast  that  assigns  probability  one  to  one  of  the  possible  outcomes  ?nd  probability 
zero  to  all  the  others  is  defined  as  an  absolute  forecast  The  vector  r  such  that 

|l  if  /  —  Ac 
r‘  "  |0  otherwise 

is  an  absolute  forecast  of  outcome  k . 

Definition  4: 

A  perfect  forecast  is  an  absolute  forecast  of  the  actual  outcome. 

Theorem  1: 

If  a  scoring  rule  is  (strictly)  proper  and  the  outcome  is  k%  the  score  for  outcome  k  is 
maximized  (uniquely)  by  a  perfect  forecast  of  k. 

Proof: 

Let  5  be  a  strictly  proper  scoring  rule,  then 
*  » 

£  S,(r)pi  <  £  S,(p)p,  V  /?,  with  equality  if  and  only  if  r  —  p  , 

/-i  /-i 

or 

n 

£  ( S,(p )  —  S,(r))pi  ^  0  V  p,  with  equality  if  and  only  if  r  —  p  . 

<-i 

Let  p  be  a  perfect  forecast  of  outcome  k. 

1  if  /  —  Ac 
0  otherwise 

then 


V 
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3.  NECESSARY  AND  SUFFICIENT  CONDITIONS  FOR  A  SCORING  RULE  TO  BE 
STRICTLY  PROPER 

Definition  5: 

A  subset  C  of  R*  is  said  to  be  convex  if  (l-\)x+\y  €  C  where  x  €  C,  y  €  C  and 
0<A<1. 

Definition  6: 

A  real-valued  Junction  f  on  a  convex  set  C  is  said  to  be  (strictly)  convex  on  C  if, 

/((l  -  \)x{  +  kx2)  (<)  <  (1  -  A)/(x,)  +  X/(x2),  0  <  X  <  1 
for  any  two  different  points  xx  and  x2  in  C. 

Definition  7: 

A  vector  x  *  is  said  to  be  a  subgradient  of  a  convex  function  /  at  a  point  jc  if, 
f(y)  ^  fix)  +  ((y  —  x),x*} 

where  ( •  ,  •  )  is  the  usual  inner  product.  This  condition  is  referred  to  as  the  subgradient  ine¬ 
quality (l).  If  /  is  strictly  convex,  equality  occurs  if  and  only  if  x—y. 

Theorem  2: 

A  scoring  rule  S  is  strictly  proper  if  and  only  if 

Skir)  -  Gir)  -  (r,G*(r))  +  Gk(r)  +  ak 
where  r  is  the  vector  of  probability  forecasts  of  dimension  /?,  k  is  the  outcome, 

G*(r)  -  (G,‘(r) . Gk(r) . G;(r)) 

is  a  subgradient  at  the  point  r  of  the  strictly  convex  function  G,  and  ak  is  a  constant  that  may 
depend  on  the  outcome  k  but  not  on  the  vector  of  probability  forecasts  r. 


(II  The  subgradient  inequality  has  a  simple  geometric  meaning  when  /  is  finite  at  X  it  says  that  (he  graph 
of  the  affine  function  h  (v)"/(x)+|(y- x),Jf  )  is  a  non-vertical  supporting  hyperplane  to  the  convex  set 
epi  /  at  the  point  (jc,/uc))  See  Rockafellar  1970.  Section  23  [Reference  21 


v 
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Proof: 

Suppose 

Sk(r )  —  G(r)  —  ( r,G'(r ))  +  Gk(r)  +  ak 

where  G  ( r )  is  a  subgradient  at  the  point  r  of  the  strictly  convex  function  G  and  ak  is  a  con¬ 
stant  that  depends  on  k  but  not  on  r. 

We  wish  to  show  that  S  is  a  strictly  proper  scoring  rule.  Let  p-(pi„..,p„)  be  the 
forecaster's  true  judgment  and  a— (a  . . a,),  then 

£S(p,p)  -  ES(r,p)  -  G(p)  -  (p.C *(/>))  +  (p,G*(p))  +  (p.a) 

-  [G(r)  -  ( r,G'(r ))  +  (p,G*(r))  +  (p,a)l 

therefore. 

ES(p,p)  -  ES(r,p )  -  G(p)  -  G(r)  -  ((p  -  r),G*(r)) . 


Since  G'(r)  is  a  subgradient  at  the  point  r  of  the  strictly  convex  function  G, 

G(p)  ^  G(r)  +  (( p  —  r),G  (/•))  Vp,  with  equality  if  an d  if  /•  —  p  . 

Therefore, 


ES(p,p)  >  ES(r,p)  Vp,  with  equality  if  and  o/r/y  if  r  •  p  . 
So  5  is  a  strictly  proper  scoring  rule. 

Conversely,  suppose  5  is  a  strictly  proper  scoring  rule,  that  is, 

£S(p,p)  >  ES(r,p )  Vp,  w///r  equality  if  and  only  if  r  «  p  . 


Let 


G(p)  -  £S(p,p)  -  (p,S(p)) . 
<?;(p)  -  S*(p>  . 
and 


then 


a*  -  0  , 

S*(p)  -  G(p)  -  (p,G*(p))  +  G*(p)  +  a*  . 


V 
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To  show  that  G  is  strictly  convex  and  G\p)  is  a  subgradient  at  the  point  p  of  G,  note  that 

G(p)  >  £S(r,p)  —  (p,S(r))  V  ps  with  equality  if  and  only  if  r  p 
since  S  is  strictly  proper. 

For  each  fixed  r,  the  function  £S(r,p),  which  is  linear  in  p,  lies  strictly  below  the  func¬ 
tion  G,  except  at  r,  where  the  linear  function  and  G  have  the  common  value  £$(/\r)-G(r). 
In  short,  £S(/\p)  is,  for  each  r,  a  linear  function  of  support  of  G  at  the  point  r  and  only  there. 
Therefore,  G  is  a  strictly  convex  function  of  p.  (This  follows  from  [21  Rockafellar  (1970), 
Theorem  5.5,  page  35.) 

Now  consider, 

G(p)-G(r)-((p-r),G‘(r)) 

-  ( p,5(p))-  (r,5(r))-  (p,S(r))  +  (r,S(r)) 

-  (p,5(p))  -  (p,S00)  -  £S(p,p)  -  £S(r,p)  >  0 

V  p,  with  equality  if  and  only  if  r-p,  since  S  is  a  strictly  proper  scoring  rule.  So, 

G(p)  >  G(r)  +  ((p  -  r),G’(r)}  Vp,  with  equality  if  and  only  if  r  •  p  . 


Therefore,  G’(r)  is  a  subgradiem  at  the  point  r  of  the  strictly  convex  function  G.® 


Corollary  1: 

A  scoring  rule  S  is  strictly  proper  if 


Sk(r )  -  G(r)  -  (r,VCW)  +  Gk(r )  +  a* 
where  r  is  the  vector  of  probability  forecasts  of  dimension  n  and  k  is  the  outcome, 


VG(r)  - 

dr, 


dG(r) 

dr. 


«M<r). 


Git(r), 


G„(r)) 


is  the  gradient  at  the  point  r  of  the  strictly  convex,  differentiable  function  G,  and  ak  is  a  con¬ 
stant  that  may  depend  on  the  outcome  k  but  not  on  the  vector  of  probability  forecasts  r. 
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Proof: 

The  gradient  is  also  a  subgradient  since 

G(p )  ^  G(r)  +  ((p  —  r)  ,  V (?(/■))  Vp,  w'r/r  equality  if  and  only  if  r  —  p  . 

Apply  Theorem  2  and  the  result  follows*  ■ 

The  results  of  this  section  are  quite  important.  They  provide  us  with  another  way  to 
recognize  strictly  proper  scoring  rules,  but  more  important  still,  they  give  us  a  method  to  con¬ 
struct  them.  In  Section  4t  the  results  of  this  section  are  used  in  the  construction  of  a  strictly 
proper  scoring  rule. 


,3' 


4.  CONSTRUCTION  OF  STRICTLY  PROPER  SCORING  RULES  BY  THE  USER.  AN 
APPLICATION 

In  Section  2,  the  Brier  Scoring  Rule  was  presented.  It  has  the  desirable  property  of  being 
strictly  proper.  However,  it  does  not  allow  for  the  user  to  be  represented  in  the  scoring  rule. 
One  way  the  user  can  be  taken  into  consideration  is  the  inclusion  of  his  utility  function  in  the 
scoring  rule. 

In  this  section,  we  construct  a  scoring  rule  that  is  a  function  of  the  user’s  utility.  The 
construction  is  based  on  Theorem  2  of  Section  3  and  it  illustrates  the  way  in  which  this 
theorem  can  be  used  to  construct  strictly  proper  scoring  rules  for  different  situations. 

4.1.  The  Model 

Suppose  there  are  n  possible  outcomes  or  events  £, . E„  and  that  the  user  of  the  fore¬ 

cast  or  decision  maker  can  take  one  of  m  possible  actions  A  t,. 

Let  uu  be  the  utility  of  the  user  when  the  action  he  took  was  A,  and  the  outcome  is  Er 
Then  there  is  a  matrix  U  of  utilities 


U  -  . 


11 


ij 


u  ,  . 

ml 


In 


am 


The  value  of  the  utility  of  each  point  of  matrix  U  is  relative  to  the  utility  of  the  other 


points.  Without  loss  of  generality  it  is  assumed  that  u,t>Q  V  ij. 


i 
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The  forecaster  attaches  a  probability  to  the  occurrence  of  each  event; 

n 

r-(r . . r„)\  where  r  is  the  vector  of  probability  forecasts. 

f-i 


4.2.  Construction  of  a  Strictly  Proper  Scoring  Rule 

Given  the  above  model,  a  strictly  proper  scoring  rule  is  constructed  based  on  Theorem  2 


of  Section  3. 


m 


(0  Let  G(r)  -  <? 
where  U,  is  row  i  of  matrix  U. 

(ii)  Let  G*(r)  -  VG(r)  —  gradient  of  G  at  r  . 


C*(r)  -  Gk(r)  -  k  -  1 . n 

0rk 


for  all  r  6  C  for  which  G  is  differentiable  where  the  set 


C  - 


«  convex  . 


Jr  |  /■€/?";  £  r,  -  1;  r,  >  0,  /  -  1 . /»J 

If  r€C  and  G  is  not  differentiable  at  the  point  r,  G'(r )  is  defined  as  the  right  (side  direc¬ 
tional)  derivative  of  G  at  the  point  r. 

G\r)  -  <7+(r)  -  <7(r.l)  -  lim  *1  Z  Qlll  . 

aio  a 

Now  G’(r)  is  well  defined  since  G  is  strictly  convex  [see  proof  below]  and,  therefore,  the 
directional  derivative  exists  for  all  r€C  [Reference  2].  Also,  G+(r)-VG(r)  if  G  is 
differentiable  at  r. 

(iii)  Let  ak  be  a  constant  that  depends  on  the  kth  outcome. 

Claim  1: 

G(r)—e  ‘  is  strictly  convex. 


X 


15' 


Proof: 


<nu(rM,) 


(r.U) 


since  e  is  increasing  in  x. 
Define 


G(r,i )  •  e''r'V‘l  i  —  1,  ....  m 


The  Hessian  H  of  G(r,i)  with  respect  to  r  follows: 


H  -  e 


2 

U11  • 
2 

.  u“ 


12 


u.,u. 
11  In 


•  u  u 
11  in 


In 


We  need  to  show  that  H  is  a  positive  definite  matrix. 

Let  y  be  an  arbitrary  vector  of  dimension  n,  then 

yrHy  -  ■  (yi^i  +  •  •  •  +  y*u,„)2 . 

Recall  that,  by  assumption,  «(>>0  for  all  ij.  Therefore,  yrHy>0  for  all  y^O,  so  H  is  posi¬ 
tive  definite.  This  implies  that  G(rj)  is  strictly  convex.  Since 

G(r)  -  max  G(rj)  , 

# 

it  follows  that  G  is  a  strictly  convex  function  of  r .■ 
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G'(r)  is  a  subgradient  at  the  point  r  of  the  strictly  convex  function  G  by  definition. 
Before  using  Theorem  2  to  construct  a  strictly  proper  scoring  rule,  the  following  set  is  defined, 

A  -  W  I  (r,U,)>  (r,Uj)  *  j  *  A  /-I . »  . 

Claim  2: 


A  >s  a  convex  set,  /->!,. 


Proof: 


rj  €  D,  implies  Vy  *  ,  • 


Then, 


x(r.  A)  +  (1  -  X)(r,  A)  >  X(r, Uj)  +  (1  -  x)<r, Uj) 
V  j  *  i  and  0  <  X  <  1  . 


So, 


(Ur  +  (1  -  X)/),  A)  >  (Ur  +  (l  -  X)/),  A) 

V  j  *  i  and  0  <  X  <  l  . 

Therefore,  A,  is  a  convex  set.* 

Now,  using  the  sets  A. 

I p  n\ 

G(r)  —  e''  '  '*  r  €  D, 

GkU)  -  uike'r'U'1  r  €  A  ;  Ir,r  +  «)  C  A, 
where  for  some  e  such  that  « k>0.  Let  the  scoring  rule  S  be  as  follows. 


S„(r) 


/ .  n\ 

e\'-uil 


Sk(r)  -  G(r)  -  (r,G’(r))  +  Gk(r)  +  ak 

(r,  U,e''r'U’)  +  uikex''Ul1  +  ak  r  €  A;  If,/’  +  «)  C  A,  • 


So, 

S*(r)  ■■  (1  -  (r,  A)  +  «<*)  +  a*  r  €  A,;  (/■,/•  +■  c)  C  A,  • 

5  is  well  suited  for  the  model  presented  in  Section  4.1.  The  score  that  the  forecaster  gets 
is  a  function  of  the  expected  utility  of  the  user  when  he  takes  action  A,  and  also  of  the  utility 
of  the  specific  outcome.  The  term  ak  depends  only  on  the  outcome  k  and  can  be  disregarded 
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while  interpreting  the  score. 

(r,U,)  is  the  expected  utility  of  the  user  given  the  forecast  r  €  Dr  u,k  is  the  utility 
obtained  from  having  taken  action  A,  when  the  outcome  is  £*.  ulk  is  the  term  that  most  clearly 
improves  the  forecaster’s  score. 


By  construction.  5  is  a  strictly  proper  scoring  rule  (Theorem  2],  The  score  for  outcome  k 
is  maximized  uniquely  by  a  perfect  forecast  of  k  [Theorem  ll. 

For  comparison,  suppose 

G(r)  -  ( r,U ,)  r  €  D, 

Gk(r)  -  u,k  r  €  D,  ;  (r.r  +  «)  C  D,  . 

Note  tfeat  G(r»  is  now  convex  (not  strictly  convex.)  Following  the  same  procedure  as  before 

ws  gei, 

Sk(r)  -  u,k  +  ak  r  6  D,  ;  [r,r  +  «)  C  D, 
which  is  a  proper  scoring  rule  (not  a  strictly  proper  scoring  rule). 


t 
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5.  SUMMARY  AND  CONCLUSIONS 

A  strictly  proper  scoring  rule  was  constructed  which  takes  into  consideration  the  utility 
function  of  the  user  of  the  forecast.  The  scoring  rule  was  presented  as  an  application  of 
Theorem  2  in  Section  3  which  gives  the  necessary  and  sufficient  conditions  for  a  scoring  rule  to 
be  strictly  proper. 

Suppose  a  model  of  a  given  situation  is  presented.  By  finding  a  strictly  convex  function 
that  accounts  for  some  part  of  the  behavior  of  the  model,  a  strictly  proper  scoring  rule  can  be 
constructed  following  the  procedure  of  Section  4.  More  applications  of  this  methodology  will 
be  presented  in  a  forthcoming  paper.  Also,  the  case  of  a  continuous  density  function  forecast 
(as  opposed  to  a  discrete  probability  forecast)  win  be  studied  separately. 

Different  scoring  rules  are  needed  because  forecasters  are  not  ideal  subjects.  In  theory, 
any  strictly  proper  scoring  rule  should  encourage  the  forecaster  to  state  his  true  beliefs.  How¬ 
ever,  this  does  not  happen  in  practice  and  the  need  arises  for  scoring  rules  adapted  to  each 
situation.  Also,  the  forecaster  being  subjected  to  evaluation  by  scoring  rules  should  have  a 
minimum  knowledge  of  probabilistic  concepts  and  of  utility  theory  in  order  to  realize  that  his 
objective  is  to  maximize  his  expected  score. 
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